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ABSTRACT 


This  thesis  explores  the  dimensionality  of  the  Defense  Language  Institute’s  (DLI) 
primary  student  feed  back  tool  -  the  Automated  Student  Questionnaire  (ASQ).  La 
addition  a  data  set  from  ASQ  2.0  (the  newest  version)  is  analyzed  for  trends  in  student 
satisfaction  across  the  sub-scales  of  sex,  pay  grade,  and  Defense  Language  Proficiency 
Test  (DLPT)  results. 

The  method  of  principal  components  is  used  to  derive  initial  factors.  Although  an 
interpretation  of  those  factors  seems  plausible,  they  are  subjected  to  a  factor  analysis 
rotation  (varimax)  and  five  factors  are  determined  and  interpreted.  Each  of  the  five 
factors  are  interpreted  in  terms  of  student  satisfaction  with  DLI’s:  (1)  academic 
environment,  (2)  military  environment,  (3)  non-barracks  dormitory  living  conditions,  (4) 
official  and  supplemental  course  audio  tapes,  and  (5)  service  unit ’s  computer  learning 
centers.  From  the  factor  loading  matrix  factor  scores  equations  are  developed  for  use  in  a 
sub-scale  trend  analysis. 

Using  non-parametric  procedures,  each  factor  is  checked  for  differences  in  central 
tendency  by  sex,  pay  grade,  and  DLPT  score  (DLPT  consists  of  three  tests  DLPTL, 
DLPTR,  DLPTS).  From  this  analysis  the  following  results  derive:  (1)  sex  has  no  effect  on 
any  of  the  factors,  (2)  pay  grade  affects  satisfaction  with  the  military  environment,  (3) 
DLPTL,  DLPTR,  and  DLPTS  affect  satisfaction  with  the  academic  environment,  and 
DLPTS  also  affects  satisfaction  with  the  computer  learning  centers. 
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INTRODUCTION 


This  chapter  provides  some  background  on  the  Defense  Language  Institute  (DLI) 
then  addresses  the  problem  this  thesis  treats  and  the  methodology  employed. 

A.  BACKGROUND 

This  section  briefly  describes  DLI’s  mission,  vision,  goals,  student  population, 
entrance  requirements,  language  assignments,  and  proficiency  standards.  The  genesis  of 
their  primary  student  feedback  tool,  the  Automated  Student  Questionnaire,  (ASQ)  is  also 
presented. 


1.  DLI  -  Mission,  Vision,  and  Goals 

DLI’s  home  page  describes  their  mission  in  the  following  way:  “To  train,  sustain, 
and  evaluate  foreign  language  skills...”  As  the  world’s  “...largest...”  and  “...most 
respected  language  resource  institute...”  their  vision  for  the  21st  Century  is  to  remain 
“...flexible  and  responsive  to  the  changing  world  order  and  on  the  cutting  edge  of  technical 
innovations...”  From  their  goals  statement,  those  germane  to  this  thesis  are  as  follows:  (1) 
Training  -  80%  of  students  achieve  level  two  proficiency  in  listening,  reading,  and 
speaking,  (2)  Evaluating  -  develop  valid  and  reliable  assessment  tools,  (3)  Quality 
Philosophy  -  create  a  culture  of  continuous  quality  improvement,  (4)  Quality  of  Life  - 
develop  and  maintain  affordable  quality  of  life  programs  that  allow  students  to  focus  on 
learning. 

2.  Student  Population 

Each  year  the  Defense  Language  Institute  trains  thousands  of  military,  DOD,  and 
civilian  personnel  in  a  wide  variety  of  foreign  languages.  The  Institute  plans  for  2,900 
students  annually  and  since  its  inception  in  1941  has  graduated  more  than  155,000.  This 
student  population  is  screened  for  a  minimum  aptitude  level  prior  to  enrollment  at  DLL 
Military  students  are  first  screened  for  minimum  language  aptitude  via  the  Armed  Services 
Vocational  Aptitude  Battery  (ASVAB).  Those  that  pass  the  ASVAB’s  language  aptitude 
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section  are  then  evaluated  with  the  Defense  Language  Aptitude  Battery  (DLAB).  The 
DLAB  is  based  on  an  artificial  language  and  is  designed  to  measure  the  student’s  ability  to 
leam  foreign  language  in  general,  not  his/her  ability  to  learn  a  specific  foreign  language. 

3.  Language  Assignment 

Students  are  assigned  to  a  specific  language  based  on  their  DLAB  score.  Once 
assigned,  the  specific  course  of  instruction  varies  in  length  and  difficulty  depending  on  the 
language.  Languages  are  categorized  on  a  difficulty  scale  ranging  from  one  to  four.  A 
category  one  language  is  a  close  cognate  of  the  student’s  native  language  (assumed  to  be 
English)  such  as  Spanish  or  French.  These  languages  have  common  words,  similar 
sounds,  sentence  structure,  and  recognizable  symbols  (letters).  In  contrast,  category  four 
languages  are  considered  “truly”  foreign  languages  and  essentially  have  nothing  in 
common  with  English.  Currently  24  languages  and  dialects  are  taught  and  they  are 
organized  into  seven  different  language  schools:  Asian  1,  Asian  2,  Middle  Eastern  1, 
Middle  Eastern  2,  East  European  1,  East  European  2,  and  West  European/Latin 
American.  These  schools  are  organized  into  departments  that  employ  the  Team  Teaching 
Concept  with  teams  of  six  instructors  responsible  for  three  groups  of  about  ten  students 
each. 


4.  Completion  of  Training 

Upon  completion  of  training,  students  are  evaluated  for  language  proficiency  by 
another  standardized  test,  the  Defense  Language  Proficiency  Test  (DLPT).  The  DLPT  is 
best  considered  as  three  separate  tests:  (1)  DLPTL,  for  “listening,”  (2)  DLPTS,  for 
“speaking,”  and  (3)  DLPTR,  for  “reading.”  For  each  of  the  three,  the  scale  on  which 
proficiency  is  recorded  is  an  eleven  point  scale  (0,  0+,  1, 1+,  2,  2+,  3, 3+,  4,  4+,  5)  on 
which  zero  represents  no  proficiency  at  all  and  five  represents  a  proficiency  level 
associated  with  a  native  speaker.  As  noted  in  the  goals  statement,  the  targeted 
achievement  level  for  the  institution  is  that  80  percent  of  the  students  score  two  or  better 
on  each  of  DLPTL,  DLPTS,  and  DLPTR.  This  targeted  achievement  level  can  be  roughly 
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equated  to  the  proficiency  level  expected  from  a  college  language  major  after  two  full 
years  of  study. 

5.  The  ASQ 

Upon  completion  of  the  training  process  the  students  are  required  to  provide  input 
as  to  their  perception  of  quality  with  regard  to  their  overall  DLI  experience.  The  data 
collection  instrument  has  traditionally  been  survey  type  forms.  These  surveys  evolved 
from  paper  and  pencil  to  their  current  fully  automated  configuration  called  the  ASQ. 

There  are  two  versions  of  the  ASQ,  ASQ  1.0  introduced  in  March  of  1993,  and  the 
current  version,  ASQ  2.0,  implemented  in  December  1995.  Files  with  ASQ  1.0  data 
contain  approximately  ten  thousand  records  while  files  with  ASQ  2.0  data  contain  fewer 
records  and  number  approximately  one  thousand.  Data  from  both  ASQ’s  are  accessible 
and  can  be  linked  to  various  other  respondent  items  such  as:  pay  grade,  class  number, 
service  component,  sex,  DLAB,  and  DLPT  score. 

The  ASQ’s  purpose  is  to  collect  student  opinion  data  on  the  overall  quality  of  their 
DLI  experience.  This  task  is  facilitated  by  dividing  the  ASQ  into  two  parts,  each  aimed  at 
collecting  data  on  one  of  two  aspects  of  the  students’  entire  DLI  experience,  these  aspects 
are  as  follows:  (1)  Instructional  Effectiveness  (IE)  and  (2)  Program  Effectiveness  (PE). 

a)  ASQ  1.0 

ASQ  1.0  is  further  divided  into  sub-categories  within  IE  and  PE.  Each  of 
these  sub-categories  attempts  to  capture  student  opinion  on  a  particular  dimension.  These 
dimensions  are  designed  on  an  intuitive  basis  and  are  not  disclosed  to  the  respondent 
during  the  data  collection  process.  Each  main  category  of  IE  or  PE  is  divided  into  five  to 
eight  sub-categories,  where  each  sub-category  is  supported  by  one  to  nine  questions.  For 
example,  one  of  the  sub-categories  under  BE  is  “Counseling  and  Assistance”  which  is 
supported  by  three  questions  which  are  shown  below: 

(1)  Went  out  of  his  or  her  way  to  help  me  learn  the  Language. 

(2)  Gave  me  helpful,  individual  feedback  when  I  needed  it. 

(3)  Was  readily  approachable  for  counseling  or  assistance  when  I  needed  it. 
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b)  ASQ2.0 


The  total  number  of  questions  and  the  number  of  sub-categories  in  both  BE 
and  PE  are  reduced  in  ASQ  2.0.  For  the  new  PE  there  are  now  fourteen  questions  that 
are  believed  to  support  only  two  broad  sub-categories:  academic  and  military.  The  new 
BE  arrangement  is  similarly  transformed. 

6.  Other  Feedback  Sources 

DLI  staffs  internal  evaluation  teams  in  the  form  of  Training  Assistance  Visits 
(TAV)  and  hosts  external  committees  to  aid  them  in  the  curriculum  review  process. 

TAVs  and  curriculum  reviews  are  expected  to  provide  information  leading  to  better 
understanding  of  root  causes  of  problems  experienced  by  students.  TAV  reports  are 
qualitative  descriptions  of  the  school’s  management  practices,  procedures,  and  their 
impact  on  the  training  program. 

Curriculum  reviews  were  conducted  only  up  to  the  fall  of  1993;  however,  staffing 
and  procedures  for  re-establishing  a  revised  version  of  the  curriculum  review  are  now 
under  discussion.  The  primary  function  of  the  curriculum  review  was  to  furnish  the 
customer  agencies  that  use  DLI  graduates  a  first  hand  look  and  direct  input  into  the 
language  programs. 

B.  PROBLEM  STATEMENT 

In  accordance  with  the  Institute’s  goals  of  “continuous  quality  improvement”  and 
maintenance  of  “valid  and  reliable  assessment  tools,”  DLI  staff  is  evaluating  student 
feedback  as  an  indicator  of  Program  and  Instructor  Effectiveness.  They  want  to  know  if 
their  ASQ  really  measures  the  underlying  phenomena  it  purports  to  measure.  Specifically, 
with  respect  to  ASQ  2.0: 

(1)  Do  ASQ  2.0’s  14  questions  support  the  two  sub-category  design? 

(2)  If  not,  what  is  ASQ  2.0’s  dimensionality  and  which  questions  represent  those 
dimensions? 
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Additionally,  DLI’s  Research  and  Analysis  Division  would  also  like  to  see  some 
analysis  of  the  ASQ  2.0  responses  to  determine  if  student  satisfaction  levels  appear  to 
depend  on  pay  grade,  sex,  or  DLPT  scores. 

C.  METHODOLOGY 

First  the  ASQ  2.0  data  are  analyzed  to  determine  the  number  of  separate 
underlying  mathematical  factors  that  account  for  the  variability  in  the  data  set,  in  other 
words,  to  determine  the  mathematical  dimensionality.  This  is  accomplished  using  the 
method  of  principal  components.  Once  the  principal  components  are  determined,  a  factor 
analysis  is  performed  in  an  attempt  to  improve  the  interpretations  of  the  components  or 
factors.  The  number  of  factors  are  then  compared  to  the  number  of  dimensions  believed 
to  be  designed  into  the  questionnaire.  The  dimensional  design  was  not  disclosed  to  the 
author  until  after  the  factor  analysis  was  completed  and  interpretations  made.  The 
mathematical  factors  are  qualitatively  analyzed  to  examine  whether  they  make  intuitive 
sense  as  well. 
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H.  DIMENSIONALITY  ANALYSIS 


This  chapter  presents  an  analysis  of  the  ASQ  2.0  PE  data.  The  14 
questions  (variables)  are  first  analyzed  for  pairwise  correlation  followed  by 
principal  components  analysis.  Several  methods  for  eliminating  components  are 
discussed;  then,  using  a  composite  of  those  methods  seven  of  the  fourteen 
principal  components  are  eliminated  and  five  of  the  remaining  seven  are 
interpreted.  Various  numbers  of  components  are  retained  and  subjected  to  the 
varimax  factor  analysis  rotation  scheme  and  subsequent  interpretation  of  the 
rotated  factors  is  presented.  The  varimax  factor  analysis  rotation  maximizes  the 
sum  of  the  squared  correlations  for  retained  components  thereby  producing  a 
unique  rotation  matrix  which,  when  multiplied  by  the  original  matrix  of 
eigenvectors,  produces  a  unique  factor  matrix. 

A.  THE  DATA 

As  stated  in  the  introduction,  the  variables  of  this  data  set  are  the  responses 
to  questions  concerning  program  effectiveness  at  DLI,  and  the  respondents  are 
DLI  students. 

1.  The  Variables 

There  are  fourteen  variables  and  the  response  to  each  is  an  integer  in  the 
range  0-4.  The  responses  represent  the  students’  opinions  on  various  aspects  of 
their  entire  DLI  experience.  For  each  question  the  possible  responses  are: 

(1)  “0”  No  opinion  or  not  applicable 

(2)  “1”  Strongly  Disagree 

(3)  “2”  Disagree 

(4)  “3”  Agree 

(5)  “4”  Strongly  agree 

All  questions  are  worded  so  that  the  higher  number  response  relates  to  the 
question  in  a  positive  way,  e.g.,  “The  course  objectives  were  clearly  explained  to 
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me  early  in  the  program.”  rather  than,  “The  course  objectives  were  not  well 
explained”  Therefore,  for  all  questions  a  higher  number  is  interpreted  as  better 
than  a  lower  number.  The  entire  set  of  questions  is  shown  in  Figure  1 . 


1.  The  overall  instructional  program  for  my  language  was  well  organized. 

2 .  The  overall  program  goals  and  related  requirements  were  clearly  explained. 

3.  Teaching  team  cooperation  fostered  an  effective  learning  environment 

4.  The  grading  system  was  clearly  explained  early  in  the  program. 

5.  The  official  course  tapes  had  good  sound  quality. 

6.  The  official  text  materials  were  useful 

7.  Course  tests  appropriately  measured  my  ability. 

8.  Supplemental  (instructor-prepared)  language  tapes  had  good  sound  quality. 

9.  My  service  unit ’s  computer  learning  center  was  helpful  in  supporting  my  study 
program. 

10.  My  barracks/dormitory  was  comfortable  and  well  maintained. 

11.  My  government  housing  was  comfortable  and  well  maintained. 

12  The  quality  of  the  food  in  the  dining  facilities  was  good. 

13.  My  military  training  at  DU  contributed  to  my  overall  development 

14.  The  care  I  received  at  the  Troop  Medical  Clinic  was  responsive  to  my  needs. 


Figure  1:  ASQ  2.0  Questions 


2.  The  Respondents 

The  set  of  respondents  for  this  study  consists  of  615  DLI  students.  All 
students  are  unique  in  that  each  contributes  only  one  set  of  responses  to  the  data 
base.  The  students  in  the  data  base  are  summarized  by  pay  grade  and  sex  in  Figure 
2.  The  vertical  axis  contains  pay  grade-sex  combinations  and  the  horizontal  axis  is 
the  number  of  respondents. 
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Figure  2:  Number  of  Respondents  by  Subsample 
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B.  VARIABLE  CORRELATIONS 


Correlations  between  pairs  of  variables  are  computed  from  the  615 
observations  on  each  question  and  are  shown  below  in  Table  1.  Combinations  that 
appear  to  be  “reasonably”  correlated  are  shown  in  boldface  type. 


Ql 

Q2 

Q3 

Q4 

Q5 

Q6 

Q7 

Q8 

Q9 

Q10 

Qll 

Q12 

Q13 

Q14 

Ql 

1.00 

.530 

.425 

332 

.251 

353 

371 

.186 

.105 

.072 

.025 

.139 

.167 

.085 

Q2 

1.00 

363 

352 

.185 

309 

.291 

.083 

.133 

.079 

.069 

.126 

.170 

.088 

Q3 

1.00 

.265 

.137 

.233 

.273 

.175 

.089 

.035 

.027 

.006 

.045 

.079 

Q4 

1.00 

.154 

.240 

.269 

.094 

.024 

.051 

.004 

.066 

.089 

.073 

Q5 

1.00 

.185 

00 

396 

.119 

-.06 

.063 

.036 

.104 

.067 

Q6 

1.00 

303 

.146 

.049 

.081 

.055 

.123 

.118 

.139 

Q7 

1.00 

.096 

.059 

-.02 

-.04 

.037 

.084 

.088 

Q8 

1.00 

.017 

.046 

.071 

.061 

.030 

.126 

Q9 

1.00 

.260 

-.04 

.172 

.202 

.087 

Q10 

1.00 

-.34 

397 

325 

.181 

Qll 

1.00 

-.005 

.009 

.039 

Q12 

1.00 

342 

.220 

Q13 

1.00 

.286 

Q14 

1.00 

Table  1:  New  ASQ  correlation  coefficients 


From  this  correlation  matrix  several  possible  groupings  of  variables  are 
readily  apparent.  Most  notable  is  the  group  that  contains  Ql,  Q2,  Q3,  Q4,  Q6, 
and  Q7.  Two  other  groups  that  are  less  well  correlated  are  Q5  with  Q8,  and  the 
group  consisting  of  Q10,  Q12,  and  Q13 

C.  PRINCIPAL  COMPONENTS 

Principal  component  analysis  is  a  technique  for  transforming  a  set  of 
original,  interrelated  variables  into  a  set  of  new  uncorrelated  variables  where  each 
component  is  a  linear  combination  of  the  original  variables  and  the  “information” 
contributed  by  each  component  is  proportional  to  its  variance  (Afifi  and  Clark 
1990).  Thus  for  N  original  variables  X  \ ,  X  2 , ... ,  X^  we  want  to  arrive  at  a  set  of 
new  variables  Cj ,  C2,...,  C n  where: 
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Cj=aljx1  +  a2jX2+..  +  alnxN 


subject  to  the  following  three  constraints: 

(1)  The  variance  of  C;  >  variance  C2>  ...  >  variance  Cn 

(2)  All  Cj  are  orthogonal 

(3)  For  all  Cj,  a2^  a22j+...+  c?1N=  1 

Prior  to  performing  the  principal  components  analysis  the  original  variables 
are  standardized  (each  is  divided  by  its  standard  deviation)  so  that  the  principal 
components  are  found  for  the  correlation  matrix  rather  than  the  covariance  matrix. 
According  to  Afifi  and  Clark  (1990)  this  simplifies  the  analysis  because:  (1)  The 
total  variance  of  the  system  is  now  equal  to  the  number  of  variables  N,  (2)  the 
correlation  between  the  ith  principal  component  and  the/*  variable  is  simply 
rij~aij VVAR  C,- ,  where  VAR  C,  is  the  eigenvalue  of  the  ith  column.  From  the 
correlation  matrix  shown  above  as  Table  1  the  principal  components  of  the  data 
are  calculated  and  shown  below  in  Table  2. 

In  Table  2  each  column  is  an  eigenvector  and  the  eigenvalue  shown 
represents  the  variation  accounted  for  by  that  eigenvector  (each  eigenvector  is  a 
principal  component).  The  principal  components  run  sequentially  (one  through 
fourteen)  from  left  to  right  in  descending  order  based  on  the  magnitude  of  then- 
eigenvalue.  The  row  labeled  “Percent”  shows  the  percent  of  variation  accounted 
for  by  that  particular  eigenvector.  For  example,  in  column  one  the  eigenvalue,  or 
total  variation  accounted  for  by  the  first  principal  component  (PC-1)  is  3.09.  The 
total  variation  is  equal  to  the  number  of  variables,  in  this  case  14,  making  both  the 
individual  and  cumulative  percentage  of  variation  accounted  for  by  PC-1  equal  to 
3.09/14  which  we  can  see  from  Table  2  is  22.08  percent  (approximately,  due  to 
rounding).  Similarly  in  Table  2,  the  row  labeled  “CumPercent”  shows  the 
percentage  of  total  variation  accounted  for  by  the  summation  of  eigenvalues  from 
column  one  through  column  fourteen.  To  calculate  the  cumulative  percentage  for 
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any  column,  the  eigenvalue  of  that  column  and  all  previous  eigenvalues  are 
summed  and  the  sum  is  divided  by  14. 

Following  Afifi  and  Clark  (1990),  variables  that  have  a  correlation 
coefficient  greater  than  .5  with  a  principal  component  are  considered  highly 
correlated  and  said  to  “load”  on  that  component.  For  the  table  of  eigenvectors 
shown  below  this  amounts  to  values  of  ay  >  .5/VVAR  C,-  (where  C  ,■  is  column  i 
from  Table  2,  and  ay  is  the  entry  at  the  intersection  of  the  i  column  and  j  row); 
these  values  are  shown  in  boldface  type  in  the  body  of  the  table.  Again,  several 
groupings  of  component-variable  combinations  are  apparent.  Based  on  Afifi  and 
Clark’s  criterion  we  see  that  variables  Ql,  Q2,  Q3,  Q4,  Q6,  and  Q7  load  on  PC-1. 
This  grouping  is  identical  to  the  most  prominent  grouping  surfacing  in  the  analysis 
of  the  correlation  matrix  of  Table  1.  In  similar  fashion,  variables  Q10,  Q12,  and 
Q13  load  on  PC-2,  which  also  parallels  the  correlation  matrix  findings. 


PC-1 

PC-2 

PC-3 

PC-4 

PC-5 

PC-6 

PC-7 

PC-8 

PC-9 

PC-10 

PC-11 

PC-12 

PC-13 

PC-14 

Eigenvalue: 

3.09 

1.88 

1.23 

1.06 

0.95 

0.83 

0.81 

0.76 

0.70 

0.66 

0.60 

0.57 

0.44 

0.41 

Percent: 

22.08 

13.41 

8.81 

7.55 

6.80 

5.94 

5.76 

5.44 

5.00 

4.71 

4.30 

4.07 

3.16 

2.95 

CumPercent: 

22.08 

35.49 

44.30 

51.85 

58.65 

64.60 

70.35 

75.80 

80.80 

85.51 

89.81 

93.89 

97.05 

100.0 

Eigenvectors: 

Ql 

0.42 

-0.15 

-0.12 

0.04 

0.07 

0.13 

-0.07 

0.07 

-0.29 

-0.09 

-0.14 

-0.22 

-0.71 

0.30 

Q2 

-0.12 

-0.15 

0.22 

0.19 

0.17 

-0.10 

-0.06 

-0.11 

-0.47 

0.53 

-0.22 

Q3 

0.32 

-0.19 

-0.16 

-0.03 

0.06 

0.46 

0.43 

0.06 

my 

0.00 

0.11 

0.60 

0.13 

-0.10 

Q4 

j 

-0.14 

-0.19 

0.12 

-0.13 

0.05 

-0.24 

-0.62 

0.55 

0.16 

0.13 

0.10 

-0.10 

-0.01 

Q5 

0.24 

-0.14 

0.35 

-0.46 

0.19 

-0.35 

-0.24 

-0.25 

-0.14 

-0.25 

-0.26 

0.34 

0.14 

0.15 

Q6 

0.33 

-0.09 

0.01 

0.08 

-0.23 

-0.26 

-0.15 

0.60 

0.41 

-0.38 

0.18 

0.13 

-0.02 

-0.12 

Q7 

0.31 

-0.18 

-0.19 

-0.01 

-0.14 

-0.53 

0.18 

0.11 

-0.21 

0.61 

0.08 

-0.08 

0.21 

0.14 

Q8 

0.19 

-0.08 

0.42 

-0.57 

-0.22 

0.37 

-0.02 

0.09 

0.07 

0.23 

0.25 

-0.36 

0.02 

-0.17 

Q9 

0.16 

0.28 

-0.01 

-0.17 

0.73 

-0.13 

0.32 

0.09 

0.38 

0.13 

0.01 

-0.10 

-0.14 

-0.12 

Q10 

0.15 

0.54 

-0.22 

-0.18 

-0.09 

|  0.20 

-0.09 

0.08 

0.09 

-0.06 

0.08 

-0.01 

0.28 

0.67 

Ql  1 

0.01 

-0.25 

my 

0.49 

0.25 

0.16 

-0.05 

0.11 

0.12 

my 

0.12 

0.03 

0.13 

0.43 

Q12 

0.20 

0.43 

0.15 

0.16 

-0.05 

0.13 

-0.41 

0.18 

-0.07 

0.42 

-0.40 

0.27 

-0.06 

-0.30 

Q13 

0.23 

0.20 

0.22 

0.05 

-0.16 

-0.05 

-0.24 

-0.38 

-0.17 

0.63 

-0.19 

Q14 

0.18 

0.26 

0.35 

0.16 

-0.41 

-0.11 

0.60 

-0.19 

0.14 

-0.14 

-0.36 

-0.08 

-0.06 

0.02 

Table  2:  Principal  components 


Also  of  interest  is  that  Q14  is  represented  by  PC-7.  Q5  and  Q8,  which 
appeared  correlated  earlier,  do  not  pass  Afifi  and  Clark’s  loading  criterion  on  any 
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component,  however  Q5  correlates  with  PC-1  at  the  r,y  value  of  .41,  and  Q8  with 
PC-3  at  an  Vy  of  .46,  both  just  slightly  below  the  cutoff  level. 

1.  Component  Consistency 

Before  performing  any  further  analysis  on  the  principal  components, 
verification  that  the  apparent  structure  really  exists  must  be  obtained.  If  the 
components  really  exist  then  they  should  also  be  apparent  when  the  data  are 
divided  into  smaller  or  arbitrary  sub-sets.  To  check  the  consistency  of  the 
underlying  structure  the  data  are  divided  using  three  separate  schemes:  (1)  time, 
(2)  number  of  records,  and  (3)  random  sort  (each  will  be  described  in  a  separate 
paragraph  to  follow).  To  obtain  “pictures”  of  the  data  structure  GH1  biplots  are 
produced  for  the  data  reference  set  and  sub-sets.  Biplots  project  higher 
dimensional  spaces  onto  lower  dimensional  sub-spaces:  in  our  case  the  fourteen 
dimensional  data  space  is  projected  onto  a  three  dimensional  sub-space.  GH1 
biplots  scale  the  principal  components  so  that  the  standard  deviation  of  each  is 
equal  to  one.  They  are  better  at  illustrating  the  variable  correlations  than  are  the 
standard  or  “JK”  biplots.  The  rays  numbered  one  through  fourteen  represent  the 
original  variables  and  clusters  of  rays  indicate  groups  of  variables  that  are  highly 
correlated.  All  the  GH'  biplots  shown  below  give  views  from  the  same  perspective 
for  ease  of  comparison. 

a)  The  Reference  Picture 

The  components  and  rays  derived  from  the  entire  data  set  are  used 
to  produce  the  reference  picture  given  by  the  GH1  biplot  in  Figure  3.  The 
variables’  three  major  groupings  are  circled  and  labeled  in  Figure  3  below.  In  the 
following  comparisons  the  sub-set  biplots  are  compared  to  this  original  picture 
with  respect  to  cluster  composition  and  orientation  on  the  X,  Y,  and  Z  axes.  In  all 
cases  the  clusters  orient  as  follows:  (1)  group  one  points  down  and  to  the  right  and 
consists  of  Ql,  Q2,  Q3,  Q4,  Q6,  and  Q7,  (2)  group  two  points  down  and  to  the 
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left  and  consists  of  Q5,  Q8,  and  Q1 1,  (3)  group  three  points  up  in  the  direction  of 
the  Y  axis  and  consists  of  Q9,  Q10,  Q12,  Q13,  and  Q14. 


b)  Division  by  time 

For  a  first  look  at  an  arbitrary  division  of  the  data,  the  median  ASQ 
completion  date  is  chosen  as  the  dividing  line.  This  division  puts  a  total  of  223 
records  in  sub-set  one,  and  the  remaining  392  records  in  sub-set  two.  GH1  Biplots 
of  the  two  sub-sets  are  shown  below  in  Figure  4  with  sub-set  one  as  Figure  4a  and 
sub-set  two  as  Figure  4b.  It  is  apparent  that  both  divisions  of  the  data  have 
roughly  the  same  shape  as  the  reference  biplot.  In  Figure  4a  we  see  the  same  three 
groupings  of  rays  but  with  slightly  more  “distance”  among  rays  within  groups  as 
compared  to  Figure  4b;  this  greater  variability  may  be  caused  by  the  smaller 
number  of  records  allocated  to  sub-set  one. 
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a  b 


Figure  4:  GH'  Biplots  of  data  divided  by  time 


c)  Dividing  data  set  in  half 

The  second  arbitrary  grouping  scheme  divides  the  original, 
unordered  data  in  half  (approximately)  with  307  records  in  sub-set  one  (Figure  5a) 
and  308  records  in  sub-set  two  (Figure  5b).  Again,  GH'  biplots  are  presented 
below  and  we  see  the  familiar  shape  of  the  reference  data  set  reflected  in  both 
Figure  5  a  and  Figure  5b  lending  more  credence  to  the  existence  of  a  consistent 
underlying  structure. 
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Figure  5:  GIF  Biplots  of  Data  Divided  in  Half 


d)  Random  Division 

As  a  final  check  for  consistency,  the  “date  ordered”  data  are 
randomly  separated  into  two  groups.  This  separation  was  effected  using  the  Excel 
RAND  function.  The  division  scheme  is  performed  several  times  with  a  set  of 
biplots  created  and  analyzed  for  each  distinct  data  division.  All  divisions  evidenced 
similar  shapes  therefore  only  one  set  of  biplots  is  presented  below  in  Figure  6. 
Again,  the  familiar  shape  of  the  reference  data  set  is  readily  apparent  in  the 
randomly  divided  data. 
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Figure  6:  GET  Biplots  of  data  divided  randomly 


e)  Conclusions  on  Consistency 

With  three  separate  data  division  schemes  yielding  very  similar 
results,  the  data’s  underlying  structure  is  determined  to  be  consistent  and  further 
analysis  is  justified. 

2.  Component  Retention 

One  objective  of  principal  component  analysis  is  reduction  of 
dimensionality.  If  reduction  is  to  be  realized  certain  components  must  be  culled. 
Many  criteria  for  rejecting  (or  keeping)  components  exist  and  the  ones  selected  for 
this  analysis  result  from  work  done  by  Kaiser(1960),  Cattell(1966),  Afifi  and 
Clark(1990)  Jolliffe(1972),  and  Dunteman  (1989).  Kaiser  (1960)  recommends,  for 
correlation  matrices,  dropping  components  with  eigenvalues  less  than  one.  Jolliffe 
(1972)  claims,  based  on  simulation  studies,  that  Kaiser’s  “less  than  one”  criteria  is 
too  restrictive  and  often  throws  away  much  information  and  that  for  a  correlation 
matrix  .7  is  a  better  target  value  for  eigenvalues.  Cattell  (1966)  recommends  a 
“scree”  graph.  In  the  scree  graph,  eigenvalues  are  plotted  in  decreasing  order  and 
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connected  by  lines.  A  point  “k”  is  chosen  where  the  connecting  lines  are  “steep” 
to  the  left  of  k  and  “not  steep”  to  the  right  of  k;  finally  k  principal  components  are 
selected.  Afifi  and  Clark  (1990)  as  well  as  Dunteman  (1989)  recommend  retaining 
enough  components  to  account  for  a  significant  cumulative  percentage  of 
variation;  in  the  examples  shown  they  used  80  percent  as  a  guide. 

Referring  to  Table  2,  the  above  criteria  pose  several  plausible 
combinations,  all  yielding  similar  results.  Following  Jolliffe  (1972),  Afifi  and  Clark 
(1990),  and  Dunteman  (1989)  we  select  nine  principal  components  as  the  ninth 
eigenvalue  is  .7006  and  the  cumulative  variation  accounted  for  by  nine 
eigenvectors  is  80.8034  (under  PC-9  in  Table  2).  Selecting  according  to  Kaiser 
(1960)  we  choose  only  four  principal  components  where  the  fourth  eigenvalue  is 
1.06,  (the  fifth  is  .95)  and  the  cumulative  variation  accounted  for  by  four 
eigenvectors  is  51.8535  (under  PC-4  in  Table  2).  Cattell’s  scree  technique  results 
in  the  graph  shown  in  Figure  7. 


1  2  3  4  5  6  7  8  9  10  11  12  13  14 

Principle  Components 

Figure  7:  Scree  Graph  of  Eigenvalues 

The  slope  of  the  scree  graph  decreases  sharply  between  principal 
components  1  and  3,  and  then  less  sharply  between  3  and  6  and  appears  to 
decrease  at  a  constant  rate  through  the  fourteenth  component.  Based  on  the  scree 
analysis  six  components  seem  to  be  a  reasonable  choice.  The  eigenvalue  and 
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cumulative  percentage  of  variance  for  six  components  are  .8318  and  64.59 
respectively  (under  PC-6  in  Table  2).  Using  the  selection  methods  discussed 
above,  a  range  of  four  to  nine  principal  components  is  selected  for  possible 
retention.  Given  this  range  seven  components  are  chosen  for  the  following 
reasons: 

a)  Eigenvalue  and  Cumulative  Variation 

The  eigenvalue  and  cumulative  variation  for  the  seventh  component 
(PC-7  in  Table  2)  are  .8062  and  70.3547  respectively;  therefore,  its  eigenvalue  is 
well  within  the  range  between  Kaiser’s  1.00  and  Jolliffe’s  0.7  and  together  with  the 
first  six  components  accounts  for  over  70  percent  of  the  variation. 

b)  Interpretability 

Since  the  choice  of  criterion  for  component  retention  is  somewhat 
arbitrary,  it  seems  reasonable  to  select,  from  within  the  aforementioned  range  of 
four  through  nine  components,  the  number  of  components  that  seems  to  yield  the 
best  interpretation.  Again  that  number  is  seven  and  the  interpretation  of  those 
seven  components  follows. 

3.  Interpretation  of  Retained  Components 

Seven  principal  components  are  retained  for  the  reasons  stated  above.  In 
the  context  of  the  original  variables  the  following  interpretations  are  presented: 

a)  Component  One 

An  excerpt  from  Table  2  (shown  below  as  Table  3)  demonstrates 
that  variables  Ql,  Q2,  Q3,  Q4,  Q6,  and  Q7  are  all  highly  correlated  (<%  > 
.5/VVARC,  )  with  PC-1. 
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Eigenvalue: 

3.09 

Percent: 

22.08 

CumPercent: 

22.08 

Eigenvectors: 

Q1 

0.42 

Q2 

0.39 

Q3 

0.32 

Q4 

0.31 

Q5 

0.24 

Q6 

0.33 

Q7 

0.31 

Q8 

0.19 

Q9 

0.16 

Q10 

0.15 

Q11 

0.01 

Q12 

0.20 

Q13 

0.23 

Q14 

0.18 

Table  3 :  Excerpt  from  Table  2 


From  Figure  1  we  see  that  the  questions  listed  above  all  have  to  do 
with  the  academic  portion  of  the  students’  DLI  experience.  Q1  and  Q2  concern 
the  overall  instructional  program’s  organization,  goals,  and  related  requirements. 
Q4  and  Q7  are  about  the  grading  system  and  the  tests.  Q3  is  about  teaching  team 
cooperation  and  Q6  about  official  text  materials.  Therefore  PC-1  is  interpreted  as 
a  measure  of  satisfaction  with  students  ’  academic  environment. 

b)  Component  Two 

Again,  in  Table  4  we  see  that  Q10,  Q12,  and  Q13  are  highly 
correlated  with  PC-2. 
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Eigenvalue: 

Percent: 


1.88 


13.41 


CumPercent: 

35.49 

Eigenvectors: 

Q1 

Q2 

-0.12 

Q3 

-0.19 

Q4 

-0.14 

Q5 

-0.14 

Q6 

-0.09 

Q7 

-0.18 

Q8 

-0.08 

Q9 

0.28 

Q10 

Q11 

-0.25 

Q12 

0.43 

Q13 

Q14 

0.26| 

Table  4:  Excerpt  from  Table  2 


As  seen  in  Figure  1  the  questions  that  load  on  PC-2  are  military 
aspects  of  a  student’s  DLI  experience.  Q10  deals  with  barracks  living  conditions, 
Q12  with  food  in  the  dining  facility,  and  Q13  with  military  training  while  at  DLI. 
Therefore  PC-2  is  interpreted  to  be  a  measure  of  satisfaction  with  students’ 
military  environment. 

c)  Component  Three 

PC-3  appears  to  be  highly  correlated  with  only  one  question. 
However,  that  question  lends  itself  to  a  unique  interpretation  in  that  it  relates  to 
government  housing  as  opposed  to  barracks  or  dormitories.  Since  only  one 
variable  loads  heavily  on  PC-3  no  excerpt  from  Table  2  is  shown  here,  however 
the  variable’s  ay  score  is  .5804.  Since  only  this  question  is  highly  correlated  to 
PC-3  and  the  question  concerns  a  somewhat  unique  aspect  applicable  to  a  smaller 
percentage  of  students,  PC-3  is  interpreted  to  be  a  measure  of  satisfaction  with 
students  ’  non-barracks/dormitory  government  quarters. 
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d)  Components  Four  and  Six 


PC-4  and  PC-6  have  no  variables  that  load  on  them  at  the  a,y  > 

.5/VVAR  C;  level  that  have  not  already  been  used  for  a  component  with  a  higher 
overall  variance.  For  instance,  Q1 1  loads  on  PC-4  at  just  about  the  .5  level; 
however,  Q1 1  has  already  been  allocated  to  PC-3.  PC-4  does  have  weak  negative 
correlations  with  Q5  and  Q8  which  appeared  to  be  somewhat  correlated  in  the 
original  variables  correlation  matrix  shown  in  Table  1.  This  weak  negative 
relationship  may  play  a  part  in  the  factor  analysis  phase  where  it  will  be  seen  that 
Q5  and  Q8  appear  to  form  their  own  rotated  component  with  an  interpretation 
concerning  audio  tapes. 

e)  Components  Five  and  Seven 

PC-5  has  one  variable  that  is  highly  correlated:  Q9.  Q9  deals  with 
the  respondent’s  service  unit’s  computer  learning  center.  Similarly,  Q14  singularly 
loads  on  PC-7.  Q14  deals  with  the  care  the  respondent  received  at  the  Troop 
Medical  Clinic.  Therefore,  component  five  is  interpreted  as  a  measure  of 
satisfaction  with  students  ’  service  unit’s  computer  learning  center  and  seven  is 
interpreted  a  measure  of  satisfaction  with  healthcare. 

D.  FACTOR  ANALYSIS 

The  purpose  of  factor  analysis  is  to  find  new  factors  that  are  easier  to 
interpret  in  the  context  of  the  original  variables  (Afifi  and  Clark  1990).  Dunteman 
(1989)  points  out  that  if  a  suitable  interpretation  can  be  made  from  the  original 
principal  components  then  there  is  no  need  to  rotate  them.  Nevertheless,  for 
possible  enhancements  to  our  interpretation  a  factor  analysis  is  performed.  For  our 
initial  factors  we  restrict  attention  to  the  same  four  through  nine  principal 
components  chosen  in  the  previous  section.  Various  numbers  of  these  principal 
components  are  subjected  to  the  varimax  rotation  scheme  to  see  if  factor  rotation 
contributes  anything  to  our  understanding  of  the  data’s  underlying  factor  structure. 


22 


The  varimax  rotation  maximizes  the  sum  of  the  squared  factor  loadings  of  the 
retained  components,  producing  a  unique  rotation  matrix  that  when  multiplied  by 
the  original  matrix  of  eigenvectors  produces  a  unique  factor  matrix.  The  objective 
is  to  produce  factors  that  have  some  loadings  near  one  and  some  near  zero, 
thereby  making  them  easier  to  interpret.  The  criteria  for  a  successful  factor 
analysis  is  taken  from  Johnson  and  Wiehert  (1982)  and  is  known  as  the  “Wow” 
criteria: 

If,  while  scrutinising  the  factor  analysis,  the  investigator  can 
shout  Wow,  I  understand  these  factors!’,  the  application  is  deemed 
successful. 

As  is  seen  in  the  following  interpretation  the  “Wow”  criteria  is  satisfied  and  the 
rotation  adds  to  the  understanding  of  the  data’s  structure. 

1.  Interpretation  of  Rotated  Components 

The  varimax  rotation  scheme  is  applied  to  the  candidate  range  of 
components  four  through  nine.  A  discussion  and  interpretation  of  each  is 
presented  below  followed  by  a  conclusion  concerning  the  rotation  process. 

a)  Rotating  Four  Components 

When  four  components  are  rotated  the  rotation  matrix  groups  the 
questions  into  four  distinct  groups  that  differ  only  slightly  from  the  original 
principal  component  structure.  As  with  the  original  principal  components  analysis, 
Ql,  Q2,  Q3,  Q4,  Q6,  and  Q7  are  grouped  together  in  the  rotated  component  one 
(RC-1).  The  interpretation  of  this  component  remains  the  same  as  in  the  principal 
component  interpretation:  a  measure  of  satisfaction  with  students  ’  academic 
environment. 

RC-2  contains  all  of  the  questions  contained  in  PC-2  with  the 
addition  of  Q14.  This  seems  to  make  sense  as  Q14  pertains  to  adequacy  of  care 
received  at  the  Troop  Medical  Clinic.  PC-2  was  originally  interpreted:  a  measure 
of  satisfaction  with  a  students  ’  military  environment.  The  Troop  Medical  Clinic 
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is  a  military  activity  and  therefore  its  addition  to  RC-2  serves  to  strengthen  its 
original  interpretation.  Therefore  RC-2’s  interpretation  is  the  same  as  PC-2’s 
interpretation  given  above. 

RC-3  also  follows  the  original  principal  component  interpretation:  a 
measure  of  satisfaction  with  students  ’  non-barracks/dormitory  government 
quarters.  Q1 1,  which  deals  exclusively  with  government  housing,  is  highly 
correlated  with  this  rotated  component. 

Unlike  PC-4,  for  which  no  clear  interpretation  was  discernible,  RC- 
4  is  highly  correlated  with  Q5  and  Q8.  Q5  and  Q8  both  deal  with  course  (and 
supplementary,  instructor  prepared)  tapes’  audio  quality:  additionally,  these  are  the 
only  questions  that  deal  with  course  tapes.  Therefore  grouping  these  two 
questions  together  seems  to  add  to  our  understanding  of  the  data’s  factor 
structure;  RC-4  is  interpreted  as  a  measure  of  students  ’  satisfaction  with  the  audio 
quality  of  the  official  and  supplemental  course  tapes. 

Rotating  four  components  suggests  four  convenient  groupings  of 
the  variables  but  it  should  be  mentioned  that  the  four  component  rotation  scheme 
leaves  out  one  variable.  Q9,  concerning  the  students’  service  unit’s  computer 
learning  center,  is  not  strongly  correlated  with  any  of  the  four  rotated  components. 

b)  Rotating  five  Components 

When  five  original  components  are  rotated,  the  first  four  rotated 
components  are  loaded  on  by  the  same  variables  as  in  the  four  component  scheme 
discussed  above,  and  the  magnitude  of  the  loadings  do  not  change  appreciably. 

The  major  change  resulting  from  the  additional  rotated  component  is  that  Q9,  left 
out  in  the  four  component  scheme,  is  now  included  and  turns  out  to  be  highly 
correlated  with  RC-5.  Therefore,  the  interpretation  of  the  first  four  rotated 
components  remains  the  same  as  in  the  four  component  scheme  and  RC-5  is 
interpreted  to  be  a  measure  of  students  ’  satisfaction  with  their  service  unit’s 
computer  learning  center. 
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c)  Rotating  Six  through  Nine  Components 

The  effect  of  adding  a  sixth  component  for  rotation  is  to  separate 
two  variables  from  RC-1.  Q6  and  Q7  are  highly  correlated  with  RC-6.  Q6  is  as 
follows:  “The  official  text  materials  were  useful.”  Q7  reads  “Course  tests 
appropriately  measured  my  abilities.”  The  author  can  see  no  inherent  similarity  in 
these  two  questions.  The  usefulness  of  text  materials  and  measurement  abilities  of 
tests  do  not  seem  to  be  intuitively  related.  Grouping  these  two  questions  together 
-  separate  from  the  other  questions  concerning  the  overall  academic  environment  - 
seems  to  add  an  air  of  ambiguity  to  the  notion  of  an  underlying  structure.  Similar 
results  occur  from  the  addition  of  more  rotated  components.  A  seventh 
component  strips  Q14  from  RC-2  and  isolates  the  variable  as  RC-7.  This  reverses 
the  improvement  in  RC-2  over  PC-2.  Adding  an  eighth  rotated  component 
captures  Q4  as  that  eighth  component,  further  detracting  from  the  integrity  of  RC- 
1 .  Finally,  a  ninth  component  sets  apart  the  grouping  of  Q5  and  Q8. 

2.  Discussion 

The  factor  analysis  indicates  that  ASQ  2.0  is  essentially  five  dimensional  as 
compared  to  the  two  dimensions  it  is  designed  to  cover.  Those  five  dimensions 
are: 

(1)  RC-1 :  a  measure  of  satisfaction  with  students  ’  academic 

environment. 

(2)  RC-2:  a  measure  of  satisfaction  with  students  ’  military 

environment 

(3)  RC-3 :  a  measure  of  satisfaction  with  students  ’  non¬ 

barracks/dormitory  government  quarters. 

(4)  RC-4:  a  measure  of  students  ’  satisfaction  with  the  audio  quality  of 

the  official  and  supplemental  course  tapes. 

(5)  RC-5 :  a  measure  of  students  ’  satisfaction  with  their  service  unit’s 

computer  learning  center 
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For  each  of  the  five  factors  selected,  quantitative  measures  are  developed 
for  use  in  further  analysis.  These  quantitative  measures  are  called  “factor  scores.” 
Afifi  and  Clark(1990)  point  out  that  “it  is  conceivable  to  construct  factor  score 
equations  in  an  infinite  number  of  ways.”  The  factor  scores  chosen  here  are  the 
original  variable  magnitudes  scaled  by  the  factor  loading  coefficient  of  that  variable 
on  that  factor.  This  allows  the  variables  with  the  highest  loadings  to  contribute  a 
greater  percentage  to  the  total  factor  score.  The  factor  score  equations  are  given 
below  where  factor  one  is  labeled  “RC-1”  and  the  remainder  accordingly: 

(1)  RC-1  =  .753349(7/  +  .724384(72  +  .631647(75  +  ^ 6263 9 1(74  + 

.557607(76+  .629271(77 

(2)  RC-2  =  .53546(7/0  +  .70461(7/2  +  .70036(7/5  +  .67486(7/4 

(3)  RC-3  =  .914917(7// 

(4)  RC-4  =  .71238(75  +  .832315 Q8 

(5)  RC-5  =  .8491720* 
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m.  FACTOR  CENTRAL  TENDENCY  ANALYSIS 


This  chapter  presents  analysis  investigating  the  effects  of  sex,  military  rank, 
and  DLPT  score  on  student  satisfaction.  DLI’s  Research  and  Analysis  Division  is 
interested  in  trend  analysis  across  various  sub-scales;  specifically,  questions 
whether  student  sex,  military  rank,  or  DLPT  score  affect  student  satisfaction 
levels  are  considered  here.  The  rotated  component  scores  developed  in  Chapter  II 
are  used  as  quantitative  measures  of  student  satisfaction. 

A.  NON-PARAMETRIC  METHODOLOGY 

Effects  of  sex,  pay  grade,  and  DLPT  score  on  student  satisfaction  are 
explored  using  factor  scores  as  response  variables.  Some  of  the  groupings  have 
small  numbers  of  observations  associated  with  them,  making  accurate 
distributional  determinations  unlikely;  therefore,  non-parametric  comparisons  of 
central  tendency  are  employed,  avoiding  the  normality  assumption  necessary  for 
standard  techniques.  The  data  generation  mechanism  (ASQ’s  taken  from  seven 
different  schools  with  small  groups  of  students  taught  by  various  teaching  teams  of 
six  instructors  that  are  not  deliberately  kept  intact  from  one  class  to  the  next) 
provides  a  defensible  guard  against  serial  and  auto-correlation;  however,  the 
random  sampling  hypothesis  is  untenable  and  this  analysis  suffers  to  the  extent  that 
it  is  violated. 

1.  Tests  Employed 

The  object  of  this  analysis  is  to  determine  if  a  factor’s  central  tendencies 
are  the  same  across  all  groups.  Mean  (and  median)  scores  for  all  five  rotated 
components  are  tested  using  three  different  non-parametric  tests:  (1)  Wilcoxon 
Rank  Sum  Test  (Kruskal-Wallis  with  more  than  two  groups),  (2)  Median  Test 
(number  of  points  above  median),  and  (3)  Van  der  Waerden  Test  (normal 
Quantiles).  In  addition,  the  data  are  tested  for  homoscedasticity  using  four 
different  tests:  (1)  O’Brien,  (2)  Brown-Forsythe,  (3)  Levene,  and  (4)  Bartlett.  For 
handling  cases  of  heteroscedasticity  the  Welch  Anova  is  employed. 
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2. 


Rotated  Component  Scores  of  Zero 


In  all  cases,  rotated  component  scores  of  zero  are  present.  The  extent  to 
which  they  appear  and  the  effect  they  have  on  the  analysis  varies  by  question. 
Table  5  depicts  the  incidence  rates  of  “0”  for  all  five  rotated  components. 


ROTATED  COMPONENT  SCORES  OF  ZERO 

COMPONENT 

NUMBER  OF 

NUMBER  OF 

PERCENT 

ZEROS 

RESPONSES 

OF  ZEROS 

RC-1 

3 

615 

■BfRESi 

RC-2 

33 

615 

RC-3 

391 

615_ 

RC-4 

11 

615 

RC-5 _ 

_ 256 _ 

615 

■Ensm 

Table  5:  Incidence  of  Zero  Scores  for  Rotated  Components 
a)  RC-1,  RC-2  and  RC-4 

RC-1,  RC-2,  and  RC-4  all  have  six  percent  or  fewer  zeros, 


indicating  that  the  majority  of  students  answer  the  questions  associated  with  those 
factors.  Nevertheless,  to  get  a  zero  score  for  any  of  RC-1,  RC-2  or  RC-4  a 
student  has  to  respond  “No  opinion  or  not  applicable”  to  at  least  two  questions, 
and  in  the  case  of  RC-1  that  response  must  be  provided  for  six  questions.  The 
notion  that  a  student  in  a  military-academic  environment  either  has  “no  opinion” 
about  the  school  or  the  military,  or  that  the  questions  concerning  the  functioning  of 
that  environment  are  “not  applicable”  to  the  student  seems  implausible,  and  it  is 
possible  students  select  zero  for  some  other  unknown  reason.  It  is  noted  here  that 
with  the  high  positive  response  rates  for  these  rotated  components  the  inclusion,  or 
exclusion,  of  the  zeros  matters  little,  and  since  their  inclusion  is  conservative  in 
nature  they  are  kept  in  for  this  analysis. 

b)  RC-3  and  RC-5 

RC-3  and  RC-5,  shown  in  Table  5,  have  significant  incidence  rates 
for  zero  responses.  This  is  to  be  expected  because  RC-3  and  RC-5  are  both  driven 
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by  one  question  each.  RC-3  is  uniquely  associated  with  Qll,  which  concerns  only 
non-barracks/dormitory  government  housing,  and  RC-5  concerns  the  students’ 
service  unit’s  computer  learning  center.  Since  these  questions  do  not  pertain  to  all 
students  the  notion  of  “not  applicable”  seems  plausible  here;  therefore,  the  zero 
responses  are  eliminated. 

B.  EFFECTS  OF  DLPT  SCORE  ON  STUDENT  SATISFACTION 

In  all  three  DLPT  cases,  DLPT  score  means  are  found  to  be  significantly 
different  with  respect  to  RC-1  (academic  satisfaction)  and  in  the  case  of  DLPTS  a 
significant  difference  exists  across  RC-5.  A  graphical  depiction  of  central 
tendencies  for  DLPTL,  DLPTR,  and  DLPTS  across  RC-1  is  shown  below  as 
Figure  8. 

In  Figure  8  quantile  boxes  are  shown  to  illustrate  the  response  variables’ 
spread  and  medians;  additionally,  the  jagged  lines  illustrate  the  means  and  are 
connected  to  highlight  the  upward  trend.  The  straight  horizontal  line  depicts  the 
overall  mean. 
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Figure  8:  RC-1  vs.  DLPTL,  DLPTR,  andDLPTS 
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Looking  at  Figure  8  above  it  appears  that  there  exists  an  upward  trend 
across  all  three  DLPT’s  indicating  that  higher  scoring  students  tend  to  be  more 
satisfied  with  DLI’s  academic  environment  The  test  results  for  RC-1,  across  the 
three  DLPT’s,  are  presented  below  in  Table  6.  The  /^-values  shown  in  the  table 

body  indicate  the  probability  of  obtaining,  by  random  chance  alone,  a  %  value 
greater  than  the  one  calculated,  if  in  fact  the  distributions  across  the  factor  level  all 
have  the  same  center.  P-values  below  the  .05  level  are  usually  considered  ample 
evidence  that  the  distributions  do  not  have  the  same  center.  With  the  exception  of 
the  Median  test  for  DLPTL,  all  values  are  well  below  the  .05  level.  In  all  three 
cases,  the  DLPT’s  heteroscedasticity  results  indicate  that  the  variances  are  the 
same  across  all  levels. 


RC-1 

DLPTL 

DLPIR 

DLPTS 

Kruskal-Wallis 

.0014 

.0007 

<.0001 

Median 

.1689 

.0174 

.0003 

Van  der  Waerden 

.0009 

.0012 

<0001 

Table  6:  Non-parametric  Results  for  DLPT's  vs.  RC-1 


In  addition  to  the  significant  finding  for  RC-1,  it  appears  that  students  who 
score  higher  on  DLPTS  tend  to  have  a  higher  opinion  of  their  service  unit’s 
computer  learning  center.  Test  results,  verifying  the  difference  in  means  for 
DLPTS  vs.  RC-5,  are  presented  below  in  Table  7,.  There  was  no  indication  of 
heteroscedasticity  therefore  the  Welch  Anova  was  not  calculated. 


31 


TEST 

RC-5 

Kruskal-Wallis 

.0019 

Median 

.0046 

Van  der  Waerden 

.0018 

Table  7:  Test  Results  for  DLPTS  vs.  RC-5 

Although  the  test  results  shown  in  Table  7  support  a  difference  in  means, 
small  numbers  of  observations  in  the  groups  making  up  the  “tails”  of  the  sub-scale 
DLPTS,  make  the  notion  of  a  trend  somewhat  questionable. 

C.  EFFECTS  OF  PAY  GRADE  ON  STUDENT  SATISFACTION 

It  is  noted  here  that  military  rank  is  not  the  only  possible  treatment  level 
since  data  from  GS  employees  are  also  available  in  the  data  base.  However,  RC-2, 
RC-3,  and  RC-5  all  have  direct  military  connotations:  RC-2  is  interpreted  as  a 
measure  of  satisfaction  with  students  ’  military  environment;  RC-3  is  a  measure 
of  satisfaction  with  students  ’  non-barracks/dormitory  government  quarters;  RC-_5 
is  a  measure  of  students  ’  satisfaction  with  their  service  unit’s  computer  learning 
center.  In  view  of  the  above  RC-2,  RC-3  and  RC-5  are  considered  with  only 
military  personnel;  conversely  RC-1  and  RC-4  are  strictly  academically  oriented 
and  include  the  GS  data 

The  non-parametric  tests  for  central  tendency  with  respect  to  military  rank 
indicate  that  only  RC-2  means  differ  significantly  across  among  ranks.  With 
respect  to  heteroscedasticity,  RC-2  is  the  only  factor  that  indicates  significant 
differences  in  variances  across  treatment  levels  and  the  Welch  Anova  result  of 
<.0001  still  confirms  the  difference  in  central  tendency.  As  previously  stated, 
results  for  RC-2  are  computed  without  the  GS  data  but  with  the  zero  values  in  the 
data.  To  further  investigate  the  nature  of  the  dissimilarities  among  means  for 
military  rank  two  more  variations  are  examined  and  they  are  as  follows:  (1) 
military  rank  without  the  zero  values,  (2)  military  rank  without  zeros  or  levels  with 
fewer  than  ten  observations.  With  these  modifications,  the  test  results  for  RC-2 
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continue  to  indicate  that  the  null  hypothesis  should  still  be  rejected;  therefore,  it  is 
concluded  that  significant  differences  in  satisfaction  levels  exist  for  various  ranks. 
A  graphical  display  of  the  measures  of  central  tendency  for  rank  versus  RC-2  with 
all  ranks  and  zeros  included  is  presented  below  in  Figure  9. 


Figure  9:  RC-2  versus  Military  Rank 


At  first  glance  it  appears  that  there  exist  two  general  “levels”  for  the 
means:  (1)  enlisted  mean  at  about  5.5  and  (2)  officer  mean  at  about  three. 
However,  this  is  not  necessarily  the  case.  It  is  noted  here  that  pay  grades  E8  and 
01  have  only  one  observation  each  and  that  02,  04,  05,  and  06  all  have  fewer 
than  ten  observations  each.  When  the  Tukey  Kramer  Honest  Significant 
Difference  (HSD)  test  is  employed  we  find  that  there  exist  similarities  from  officer 
to  enlisted  and  differences  within  the  enlisted  group.  The  interested  reader  is 
referred  to  Appendix  A,  the  matrix  of  HSD’s  where  positive  values  indicate  pairs 
of  means  that  are  classified  as  significantly  different  by  that  test. 
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D.  EFFECTS  OF  SEX  ON  STUDENT  SATISFACTION  LEVELS 


In  this  case  the  treatment,  sex,  has  two  nominal  levels,  male  and  female.  It 
appears  that  neither  males  nor  females  evidence  a  higher  level  of  satisfaction  for 
any  of  the  factors  -  there  are  no  significant  results  from  the  non-parametric  tests 
for  central  tendency. 


34 


IV.  CONCLUSIONS/RECOMMENDATIONS 


This  chapter  provides  conclusions  concerning  the  results  from  the  various 
analyses  performed  in  this  thesis.  Recommendations  for  possible  use  of  the  factor 
scores  and  suggestions  for  further  study  are  also  provided. 

A.  PRINCIPAL  COMPONENT  ANALYSIS 

Although  the  principal  components  found  here  are  interpretable,  it  is  not 
recommended  that  their  interpretations  be  used  to  describe  the  dimensions  in  the 
data.  The  principal  components  are  useful  for  selecting  initial  factors  for  rotation; 
however,  in  this  case,  the  rotated  components  provide  a  more  thorough  and 
intuitively  defensible  interpretation. 

B.  FACTOR  ANALYSIS 

The  factor  analysis  performed  here  is  deemed  successful,  and  the  author 
has  a  high  degree  of  confidence  in  the  interpretations  provided  for  the  five  rotated 
components  chosen.  It  is  recommended  that  the  two  dimensional  ASQ  2.0 
summary  report  currently  used  at  DLI  be  modified  to  include  the  additional 
dimensions  described  in  Chapter  II  of  this  thesis.  The  variables  that  load  on  RC-3, 
RC-4,  and  RC-5  are  better  correlated  with  those  factors  than  with  the  two  factors 
from  DLLs  original  design  (academic  and  military). 

It  is  recommended  that  a  similar  analysis  be  performed  on  the  ASQ  1.0 
data.  If  the  same  factors  are  determined  to  exist  in  that  larger  data  set,  the  issue  of 
trends  across  the  sub-scales  (sex,  pay  grade,  and  DLPT)  can  be  more  thoroughly 
explored,  due  to  the  larger  number  of  observations  that  exist  for  ASQ  1.0. 
Additionally,  this  would  allow  the  possible  combination  of  the  data  bases,  or  at 
least  a  portion  of  them,  yielding  larger  numbers  of  observations  for  any  future 
studies  of  this  nature. 

It  is  further  recommended  that  DLI  consider  modifying  its  ASQ  response 
set  to  include  a  “central  scale”  variable  such  as  “neither  agree  nor  disagree.”  This 
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may  help  eliminate  some  of  the  “0”  responses  that  may  occur  because  respondents 
falsely  believe  that  checking  “no  opinion”  is  a  center  scale  response  akin  to  the 
suggested  “neither  agree  nor  disagree.” 

C.  FACTOR  CENTRAL  TENDENCIES 

The  analysis  in  this  section  represents  a  good  start  toward  a  thorough 
investigation  into  trends  in  student  satisfaction  based  on  other  population 
characteristics.  Obviously  sex,  pay  grade,  and  DLPT  are  not  the  only  sub-scales 
worthy  of  consideration.  It  is  recommended  that  further  analysis  be  done  in  this 
area.  Other  areas  of  possible  interest  are  the  different  schools/departments,  or 
possibly  students’  services.  Exploration  of  a  similar  nature  into  these  and  other 
areas  will  allow  DLI  to  further  their  knowledge  of  their  student  population,  better 
equipping  them  to  succeed  in  their  goal  of  continuous  quality  improvement. 
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